epsilon Lexicase regression benchmarks
William La Cava
February 23, 2017
## dataset method trial mse
## airfoil :450 lasso :400 Min. : 0.0 Min. : 0.001183
## concrete:450 rand :400 1st Qu.:12.0 1st Qu.: 0.471746
## enc :450 tourn :400 Median :24.5 Median : 7.278047
## enh :450 lex :400 Mean :24.5 Mean : Inf
## housing :450 afp :400 3rd Qu.:37.0 3rd Qu.:21.837452
## tower :450 dc :400 Max. :49.0 Max. : Inf
## uball5d :450 ep-lex-s :400
## yacht :450 ep-lex-sd:400
## ep-lex-d :400
## r2 time
## Min. :-1.323e+31 Min. : 0.01
## 1st Qu.: 1.000e+00 1st Qu.: 45412.01
## Median : 1.000e+00 Median :103463.23
## Mean :-2.635e+28 Mean : 79970.78
## 3rd Qu.: 1.000e+00 3rd Qu.:118045.82
## Max. : 1.000e+00 Max. :131985.44
## NA's :1
##
## ## dataset method trial mse
## airfoil :450 lasso :400 Min. : 0.0 Min. : 0.001183
## concrete:450 rand :400 1st Qu.:12.0 1st Qu.: 0.471746
## enc :450 tourn :400 Median :24.5 Median : 7.278047
## enh :450 lex :400 Mean :24.5 Mean : Inf
## housing :450 afp :400 3rd Qu.:37.0 3rd Qu.:21.837452
## tower :450 dc :400 Max. :49.0 Max. : Inf
## uball5d :450 ep-lex-s :400
## yacht :450 ep-lex-sd:400
## ep-lex-d :400
## r2 time mse_rank
## Min. :-1.323e+31 Min. : 0.01 Min. :1
## 1st Qu.: 1.000e+00 1st Qu.: 45412.01 1st Qu.:3
## Median : 1.000e+00 Median :103463.23 Median :5
## Mean :-2.635e+28 Mean : 79970.78 Mean :5
## 3rd Qu.: 1.000e+00 3rd Qu.:118045.82 3rd Qu.:7
## Max. : 1.000e+00 Max. :131985.44 Max. :9
## NA's :1
##
## ## Warning: Removed 1 rows containing non-finite values (stat_boxplot).## notch went outside hinges. Try setting notch=FALSE.
## TableGrob (3 x 3) "arrange": 9 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange gtable[layout]
## 2 2 (1-1,2-2) arrange gtable[layout]
## 3 3 (1-1,3-3) arrange gtable[layout]
## 4 4 (2-2,1-1) arrange gtable[layout]
## 5 5 (2-2,2-2) arrange gtable[layout]
## 6 6 (2-2,3-3) arrange gtable[layout]
## 7 7 (3-3,1-1) arrange gtable[layout]
## 8 8 (3-3,2-2) arrange gtable[layout]
## 9 9 (3-3,3-3) arrange gtable[layout]Timing
Stats
## [1] "airfoil"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 3.2e-13 - - - - - -
## tourn 0.00013 2.4e-13 - - - - -
## lex 0.57427 3.3e-11 8.1e-05 - - - -
## afp 5.4e-15 3.1e-16 8.1e-05 4.1e-14 - - -
## dc 2.5e-16 2.5e-16 8.1e-11 3.8e-16 0.00051 - -
## ep-lex-s 2.5e-16 2.5e-16 2.5e-15 2.7e-16 5.3e-14 8.4e-13 -
## ep-lex-d 2.5e-16 2.5e-16 2.5e-16 2.5e-16 3.8e-15 1.3e-13 0.17792
## ep-lex-sd 2.5e-16 2.5e-16 2.5e-16 2.5e-16 2.5e-15 1.6e-14 0.57427
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 0.57427
##
## P value adjustment method: holm
## [1] "concrete"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 2.4e-07 - - - - - -
## tourn 1.5e-14 5.6e-15 - - - - -
## lex 9.1e-05 3.1e-10 2.5e-06 - - - -
## afp 2.5e-16 2.7e-16 7.8e-05 5.4e-14 - - -
## dc 2.5e-16 2.5e-16 1.6e-09 9.5e-16 0.00353 - -
## ep-lex-s 2.5e-16 2.5e-16 1.3e-11 5.5e-16 3.9e-06 0.05014 -
## ep-lex-d 2.5e-16 2.5e-16 9.0e-10 1.5e-15 0.00059 0.34236 0.53724
## ep-lex-sd 2.5e-16 2.5e-16 6.2e-13 4.2e-16 2.7e-08 0.00017 0.10381
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 0.05702
##
## P value adjustment method: holm
## [1] "enc"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 4.4e-14 - - - - - -
## tourn 6.2e-05 3.3e-14 - - - - -
## lex 6.4e-05 3.1e-14 0.99826 - - - -
## afp 1.5e-12 1.9e-15 0.00462 2.7e-05 - - -
## dc 2.5e-16 2.5e-16 9.4e-14 4.7e-16 3.4e-11 - -
## ep-lex-s 2.5e-16 2.5e-16 9.3e-15 6.3e-16 9.8e-13 0.00029 -
## ep-lex-d 2.5e-16 2.5e-16 6.6e-15 1.0e-15 1.6e-13 4.9e-07 0.99826
## ep-lex-sd 5.1e-16 2.5e-16 4.1e-14 2.6e-15 1.7e-12 0.00161 0.99826
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 0.49349
##
## P value adjustment method: holm
## [1] "enh"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 4.5e-16 - - - - - -
## tourn 0.00787 1.1e-11 - - - - -
## lex 0.00050 1.8e-15 0.21849 - - - -
## afp 5.4e-08 2.5e-13 0.02175 0.00021 - - -
## dc 2.5e-16 2.5e-16 7.2e-14 2.5e-16 1.6e-09 - -
## ep-lex-s 2.5e-16 2.5e-16 9.0e-15 2.7e-16 3.9e-12 0.00043 -
## ep-lex-d 3.0e-15 1.6e-15 1.0e-14 3.0e-15 2.7e-13 1.8e-08 0.08608
## ep-lex-sd 2.5e-16 2.5e-16 2.7e-16 2.5e-16 1.4e-15 2.0e-14 0.00012
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 0.01283
##
## P value adjustment method: holm
## [1] "housing"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 1.0e-09 - - - - - -
## tourn 1.00000 1.6e-06 - - - - -
## lex 0.32872 2.7e-06 0.92393 - - - -
## afp 0.15571 1.9e-09 0.92393 0.00244 - - -
## dc 0.05376 2.3e-10 0.29856 0.00081 1.00000 - -
## ep-lex-s 1.0e-05 1.6e-13 0.00198 6.7e-08 0.09768 0.78314 -
## ep-lex-d 2.1e-06 3.7e-13 0.00081 2.6e-08 0.01799 0.35056 1.00000
## ep-lex-sd 1.5e-05 6.2e-13 0.00081 3.4e-07 0.01572 0.22273 1.00000
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 1.00000
##
## P value adjustment method: holm
## [1] "tower"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 2.5e-16 - - - - - -
## tourn 6.9e-16 2.0e-14 - - - - -
## lex 6.8e-16 4.2e-16 0.00017 - - - -
## afp 1.4e-15 7.1e-16 0.00638 0.16381 - - -
## dc 1.7e-15 2.5e-16 1.2e-07 0.00085 1.3e-05 - -
## ep-lex-s 0.53706 2.5e-16 1.0e-14 7.9e-14 7.9e-14 4.2e-12 -
## ep-lex-d 0.53706 2.5e-16 6.2e-16 7.7e-16 1.4e-15 8.3e-16 0.34560
## ep-lex-sd 0.00638 2.5e-16 1.6e-15 6.4e-15 7.6e-15 3.7e-14 0.00638
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 0.06600
##
## P value adjustment method: holm
## [1] "uball5d"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s
## rand 1.9e-15 - - - - - -
## tourn 3.8e-16 1.0e-11 - - - - -
## lex 2.5e-16 7.1e-16 0.878 - - - -
## afp 2.5e-16 1.9e-14 1.000 0.096 - - -
## dc 2.5e-16 2.5e-16 3.9e-09 3.9e-09 1.4e-13 - -
## ep-lex-s 2.5e-16 2.5e-16 2.6e-14 4.0e-13 1.9e-15 1.0e-10 -
## ep-lex-d 2.5e-16 2.5e-16 1.1e-14 2.3e-13 7.1e-16 1.0e-10 1.000
## ep-lex-sd 2.5e-16 2.5e-16 4.8e-15 1.0e-13 7.0e-16 1.5e-11 1.000
## ep-lex-d
## rand -
## tourn -
## lex -
## afp -
## dc -
## ep-lex-s -
## ep-lex-d -
## ep-lex-sd 1.000
##
## P value adjustment method: holm
## [1] "yacht"
##
## Pairwise comparisons using Wilcoxon rank sum test
##
## data: tmp$mse and tmp$method
##
## lasso rand tourn lex afp dc ep-lex-s ep-lex-d
## rand 2.4e-12 - - - - - - -
## tourn 2.5e-16 1.4e-13 - - - - - -
## lex 1.7e-15 1.2e-11 1.000 - - - - -
## afp 3.5e-15 1.7e-13 1.000 1.000 - - - -
## dc 2.5e-16 3.0e-16 5.6e-10 1.2e-08 1.2e-08 - - -
## ep-lex-s 2.5e-16 4.1e-16 2.2e-08 8.8e-08 2.4e-07 1.000 - -
## ep-lex-d 2.5e-16 5.3e-16 3.2e-10 8.1e-10 6.2e-10 0.104 0.230 -
## ep-lex-sd 2.5e-16 5.5e-16 1.5e-07 7.9e-07 4.9e-06 1.000 1.000 0.053
##
## P value adjustment method: holm## lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-d\\
## airfoil & 2.54e-16 & 2.54e-16 & 2.54e-16 & 2.54e-16 & 2.55e-15 & 1.59e-14 & 0.574 & 0.574\\
## concrete & 2.54e-16 & 2.54e-16 & 6.24e-13 & 4.25e-16 & 2.74e-08 & 0.000166 & 0.104 & 0.057\\
## enc & 5.15e-16 & 2.54e-16 & 4.12e-14 & 2.57e-15 & 1.67e-12 & 0.00161 & 0.998 & 0.493\\
## enh & 2.54e-16 & 2.54e-16 & 2.67e-16 & 2.54e-16 & 1.41e-15 & 2e-14 & 0.000121 & 0.0128\\
## housing & 1.51e-05 & 6.2e-13 & 0.000812 & 3.4e-07 & 0.0157 & 0.223 & 1 & 1\\
## tower & 0.00638 & 2.54e-16 & 1.57e-15 & 6.39e-15 & 7.63e-15 & 3.67e-14 & 0.00638 & 0.066\\
## uball5d & 2.54e-16 & 2.54e-16 & 4.8e-15 & 1.04e-13 & 6.96e-16 & 1.55e-11 & 1 & 1\\
## yacht & 2.54e-16 & 5.46e-16 & 1.52e-07 & 7.86e-07 & 4.93e-06 & 1 & 1 & 0.053\\## % latex table generated in R 3.4.1 by xtable 1.8-2 package
## % Tue Dec 12 11:29:53 2017
## \begin{table}[ht]
## \centering
## \caption{Significance test $p$-values comparing test MSE using the pair-wise Wilcoxon rank-sum test with Holm correction for multiple comparisons. All significance tests are conducted relative to semi-dynamic $\epsilon$-lexicase (ep-lex-sd). Bold indicates $p<$ 0.05.}
## \label{tbl:wilcox}
## \begingroup\footnotesize
## \begin{tabular}{rllllllll}
## \toprule
## & lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-d \\
## \midrule
## airfoil & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.55e-15} & {\bf 1.59e-14} & 0.57 & 0.57 \\
## concrete & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 6.24e-13} & {\bf 4.25e-16} & {\bf 2.74e-08} & {\bf 1.66e-04} & 0.1 & 0.057 \\
## enc & {\bf 5.15e-16} & {\bf 2.54e-16} & {\bf 4.12e-14} & {\bf 2.57e-15} & {\bf 1.67e-12} & {\bf 1.61e-03} & 1 & 0.49 \\
## enh & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 2.67e-16} & {\bf 2.54e-16} & {\bf 1.41e-15} & {\bf 2.00e-14} & {\bf 1.21e-04} & {\bf 1.28e-02} \\
## housing & {\bf 1.51e-05} & {\bf 6.20e-13} & {\bf 8.12e-04} & {\bf 3.40e-07} & {\bf 1.57e-02} & 0.22 & 1 & 1 \\
## tower & {\bf 6.38e-03} & {\bf 2.54e-16} & {\bf 1.57e-15} & {\bf 6.39e-15} & {\bf 7.63e-15} & {\bf 3.67e-14} & {\bf 6.38e-03} & 0.066 \\
## uball5d & {\bf 2.54e-16} & {\bf 2.54e-16} & {\bf 4.80e-15} & {\bf 1.04e-13} & {\bf 6.96e-16} & {\bf 1.55e-11} & 1 & 1 \\
## yacht & {\bf 2.54e-16} & {\bf 5.46e-16} & {\bf 1.52e-07} & {\bf 7.86e-07} & {\bf 4.93e-06} & 1 & 1 & 0.053 \\
## \bottomrule
## \end{tabular}
## \endgroup
## \end{table}
## dataset method N mse_rank sd
## airfoil :9 lasso :8 Min. :50 Min. :1.640 Min. :0.1414
## concrete:9 rand :8 1st Qu.:50 1st Qu.:2.910 1st Qu.:0.8114
## enc :9 tourn :8 Median :50 Median :5.480 Median :1.1606
## enh :9 lex :8 Mean :50 Mean :5.000 Mean :1.1455
## housing :9 afp :8 3rd Qu.:50 3rd Qu.:6.625 3rd Qu.:1.3690
## tower :9 dc :8 Max. :50 Max. :8.980 Max. :2.3592
## uball5d :9 ep-lex-s :8
## yacht :9 ep-lex-sd:8
## ep-lex-d :8
## se ci
## Min. :0.0200 Min. :0.04019
## 1st Qu.:0.1147 1st Qu.:0.23060
## Median :0.1641 Median :0.32983
## Mean :0.1620 Mean :0.32554
## 3rd Qu.:0.1936 3rd Qu.:0.38907
## Max. :0.3336 Max. :0.67047
##
##
## ##
## Friedman rank sum test
##
## data: mse_rank and method and dataset
## Friedman chi-squared = 54.8, df = 8, p-value = 4.828e-09## Loading required package: coin## Loading required package: survival## Loading required package: multcomp## Loading required package: mvtnorm## Loading required package: TH.data## Loading required package: MASS##
## Attaching package: 'TH.data'## The following object is masked from 'package:MASS':
##
## geyser## Loading required package: colorspace
## $Friedman.Test
##
## Asymptotic General Symmetry Test
##
## data: mse_rank by
## method (lasso, rand, tourn, lex, afp, dc, ep-lex-s, ep-lex-sd, ep-lex-d)
## stratified by dataset
## maxT = 5.1121, p-value = 8.702e-06
## alternative hypothesis: two.sided
##
##
## $PostHoc.Test
##
## rand - lasso 9.856379e-01
## tourn - lasso 9.983731e-01
## lex - lasso 9.999510e-01
## afp - lasso 9.096883e-01
## dc - lasso 2.061212e-01
## ep-lex-s - lasso 1.559229e-02
## ep-lex-sd - lasso 1.379760e-03
## ep-lex-d - lasso 2.992708e-03
## tourn - rand 7.252481e-01
## lex - rand 8.737233e-01
## afp - rand 2.981041e-01
## dc - rand 1.105767e-02
## ep-lex-s - rand 2.312908e-04
## ep-lex-sd - rand 6.674121e-06
## ep-lex-d - rand 3.721667e-05
## lex - tourn 9.999991e-01
## afp - tourn 9.993834e-01
## dc - tourn 6.648655e-01
## ep-lex-s - tourn 1.339409e-01
## ep-lex-sd - tourn 2.091101e-02
## ep-lex-d - tourn 3.730903e-02
## afp - lex 9.923325e-01
## dc - lex 4.732915e-01
## ep-lex-s - lex 6.453383e-02
## ep-lex-sd - lex 7.675381e-03
## ep-lex-d - lex 1.559229e-02
## dc - afp 9.594898e-01
## ep-lex-s - afp 4.732915e-01
## ep-lex-sd - afp 1.339409e-01
## ep-lex-d - afp 2.061212e-01
## ep-lex-s - dc 9.923325e-01
## ep-lex-sd - dc 8.306512e-01
## ep-lex-d - dc 9.096883e-01
## ep-lex-sd - ep-lex-s 9.993834e-01
## ep-lex-d - ep-lex-s 9.999510e-01
## ep-lex-d - ep-lex-sd 1.000000e+00## $Friedman.Test
##
## Asymptotic General Symmetry Test
##
## data: mse_rank by
## method (lasso, rand, tourn, lex, afp, dc, ep-lex-s, ep-lex-sd, ep-lex-d)
## stratified by dataset
## maxT = 5.1121, p-value = 1.322e-05
## alternative hypothesis: two.sided
##
##
## $PostHoc.Test
##
## rand - lasso 9.856379e-01
## tourn - lasso 9.983731e-01
## lex - lasso 9.999510e-01
## afp - lasso 9.096883e-01
## dc - lasso 2.061212e-01
## ep-lex-s - lasso 1.559229e-02
## ep-lex-sd - lasso 1.379760e-03
## ep-lex-d - lasso 2.992708e-03
## tourn - rand 7.252481e-01
## lex - rand 8.737233e-01
## afp - rand 2.981041e-01
## dc - rand 1.105767e-02
## ep-lex-s - rand 2.312908e-04
## ep-lex-sd - rand 6.674121e-06
## ep-lex-d - rand 3.721667e-05
## lex - tourn 9.999991e-01
## afp - tourn 9.993834e-01
## dc - tourn 6.648655e-01
## ep-lex-s - tourn 1.339409e-01
## ep-lex-sd - tourn 2.091101e-02
## ep-lex-d - tourn 3.730903e-02
## afp - lex 9.923325e-01
## dc - lex 4.732915e-01
## ep-lex-s - lex 6.453383e-02
## ep-lex-sd - lex 7.675381e-03
## ep-lex-d - lex 1.559229e-02
## dc - afp 9.594898e-01
## ep-lex-s - afp 4.732915e-01
## ep-lex-sd - afp 1.339409e-01
## ep-lex-d - afp 2.061212e-01
## ep-lex-s - dc 9.923325e-01
## ep-lex-sd - dc 8.306512e-01
## ep-lex-d - dc 9.096883e-01
## ep-lex-sd - ep-lex-s 9.993834e-01
## ep-lex-d - ep-lex-s 9.999510e-01
## ep-lex-d - ep-lex-sd 1.000000e+00##
## Call:
## lm(formula = mse_rank ~ method, data = rank_stats)
##
## Residuals:
## Min 1Q Median 3Q Max
## -4.1925 -0.3806 0.0387 0.3587 1.8875
##
## Coefficients:
## Estimate Std. Error t value Pr(>|t|)
## (Intercept) 7.0725 0.2991 23.650 < 2e-16 ***
## methodrand 1.4950 0.4229 3.535 0.000771 ***
## methodtourn -0.9925 0.4229 -2.347 0.022095 *
## methodlex -0.6375 0.4229 -1.507 0.136716
## methodafp -1.6425 0.4229 -3.884 0.000249 ***
## methoddc -3.0575 0.4229 -7.229 7.95e-10 ***
## methodep-lex-s -4.3700 0.4229 -10.333 3.44e-15 ***
## methodep-lex-sd -4.7925 0.4229 -11.332 < 2e-16 ***
## methodep-lex-d -4.6550 0.4229 -11.007 2.60e-16 ***
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
##
## Residual standard error: 0.8458 on 63 degrees of freedom
## Multiple R-squared: 0.8785, Adjusted R-squared: 0.8631
## F-statistic: 56.96 on 8 and 63 DF, p-value: < 2.2e-16## Analysis of Variance Table
##
## Response: mse_rank
## Df Sum Sq Mean Sq F value Pr(>F)
## method 8 325.99 40.749 56.956 < 2.2e-16 ***
## Residuals 63 45.07 0.715
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1## Call:
## aov(formula = mse_rank ~ method, data = rank_stats)
##
## Terms:
## method Residuals
## Sum of Squares 325.9942 45.0738
## Deg. of Freedom 8 63
##
## Residual standard error: 0.845847
## Estimated effects may be unbalanced## [1] "ep-lex-s-lasso"
## [1] "{\\bf 1.55e-11}"
## [1] "ep-lex-s-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-s-tourn"
## [1] "{\\bf 1.36e-09}"
## [1] "ep-lex-s-lex"
## [1] "{\\bf 6.19e-11}"
## [1] "ep-lex-s-afp"
## [1] "{\\bf 6.32e-07}"
## [1] "ep-lex-s-dc"
## [1] "0.066"
## [1] "ep-lex-s-ep-lex-s"
## [1] NA
## [1] "ep-lex-s-ep-lex-sd"
## [1] NA
## [1] "ep-lex-sd-lasso"
## [1] "{\\bf 1.54e-11}"
## [1] "ep-lex-sd-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-sd-tourn"
## [1] "{\\bf 4.00e-11}"
## [1] "ep-lex-sd-lex"
## [1] "{\\bf 1.63e-11}"
## [1] "ep-lex-sd-afp"
## [1] "{\\bf 1.17e-08}"
## [1] "ep-lex-sd-dc"
## [1] "{\\bf 3.59e-03}"
## [1] "ep-lex-sd-ep-lex-s"
## [1] "0.98"
## [1] "ep-lex-sd-ep-lex-sd"
## [1] NA
## [1] "ep-lex-d-lasso"
## [1] "{\\bf 1.54e-11}"
## [1] "ep-lex-d-rand"
## [1] "{\\bf 1.53e-11}"
## [1] "ep-lex-d-tourn"
## [1] "{\\bf 1.05e-10}"
## [1] "ep-lex-d-lex"
## [1] "{\\bf 1.86e-11}"
## [1] "ep-lex-d-afp"
## [1] "{\\bf 4.32e-08}"
## [1] "ep-lex-d-dc"
## [1] "{\\bf 1.00e-02}"
## [1] "ep-lex-d-ep-lex-s"
## [1] " 1"
## [1] "ep-lex-d-ep-lex-sd"
## [1] " 1"## % latex table generated in R 3.4.1 by xtable 1.8-2 package
## % Tue Dec 12 11:29:57 2017
## \begin{table}[ht]
## \centering
## \caption{Post-hoc pairwise statistical tests of rankings across problems according to Tukey's Honest Significant Difference test. Bold values indicate $p<$ 0.05 with adjustment for multiple comparisons.}
## \label{tbl:hsd}
## \begin{tabular}{rllllllll}
## \toprule
## & lasso & rand & tourn & lex & afp & dc & ep-lex-s & ep-lex-sd \\
## \midrule
## ep-lex-s & {\bf 1.55e-11} & {\bf 1.53e-11} & {\bf 1.36e-09} & {\bf 6.19e-11} & {\bf 6.32e-07} & 0.066 & & \\
## ep-lex-sd & {\bf 1.54e-11} & {\bf 1.53e-11} & {\bf 4.00e-11} & {\bf 1.63e-11} & {\bf 1.17e-08} & {\bf 3.59e-03} & 0.98 & \\
## ep-lex-d & {\bf 1.54e-11} & {\bf 1.53e-11} & {\bf 1.05e-10} & {\bf 1.86e-11} & {\bf 4.32e-08} & {\bf 1.00e-02} & 1 & 1 \\
## \bottomrule
## \end{tabular}
## \end{table}Convergence
## method dataset trial gen
## rand :8400 airfoil :8400 Min. : 0.00 Min. : 0
## tourn :8400 concrete:8400 1st Qu.:12.00 1st Qu.: 250
## lex :8400 enc :8400 Median :25.00 Median : 500
## afp :8400 enh :8400 Mean :24.53 Mean : 500
## dc :8400 housing :8400 3rd Qu.:37.00 3rd Qu.: 750
## ep-lex-s :8400 tower :8400 Max. :49.00 Max. :1000
## ep-lex-sd:8400 uball5d :8400
## ep-lex-d :8400 yacht :8400
## best_fit novelty
## Min. :0.0009085 Min. :0.0010
## 1st Qu.:0.0302705 1st Qu.:0.5730
## Median :0.1385115 Median :0.8060
## Mean :0.1733712 Mean :0.6973
## 3rd Qu.:0.2447150 3rd Qu.:0.8980
## Max. :0.9668800 Max. :1.0000
##
## ## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
## Warning: position_dodge requires non-overlapping x intervals
